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ABSTRACT 

Incorporated in the nedia evaluation nodel developed 
by the Computer Based Project (Syracuse, N.Y.) is a procedure whereby 
independent saaples are used to obtain baseline responses to ~ . 
cognitive question iteas. Iteas from two filBs are interalxed and 
presented after showing only one of the fills; thus, the saaple 
audience responds to questions from both the unseen film and the film 
which was seen. This study sought to determine whether the procedure 
affects the responses of the samples. Statistical analyses are 
performed on the data gathered from educable mentally. retarded (EMR) 
and normal samples receiving mixed questions sets. This is in 
contrast to EMR and normal samples receiving questions only ou the 
film seen. The results indicate that there seems to be an effect of 
mixing^items which results in lower scores for EMR children on the 
criterion items for the film seen than when only items pertaining to 
the film were asked. One explanation for this finding is that greater 
respondent frustration can be expected from the mixing of relevant 
(film soen) and irrelevant (film not seen) items, resulting in a 
lower number of correct scores. (Author/HCM) 
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EFFECTS OF MIXED AND UNMIXED CRITERION QUESTIONS 



V4J ABSTRACT 



Incorporated In the media evaluation model developed by the Computer 
Baeed Project (Syracusej N» Y*) is a procedure whereby independent samples 
are used to obtain baseline responses to cognitive question items. Items 
from two films are Intermixed and presented after showing only one of ^ 
the films I thus, the sample audience responds to questions from both the 
unseen film and the film which was seen. This study sought to determine 
whether the procedure effects the responses of the samples/ Statistical 
analyses were performed on the data g^^therM f^^ mentally re- 

tarded (EMiR) and normal samples receiving mixed question sets and from 
EMR and normal samples receiving questions only on the film seen. The 
results indicate that there seems to be an effect of mixing items which 
results in lower scores for EMR children on the criterion items for the 
film seen than when only items pertaining to the film were asked. One 
explanation for this finding is that greater respondent frustration can 
be expected from the mixing of relevant (film seen) end irrelevant (film 
not seen) items, resulting in a lower number of correct scores, 
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SPECIAL REPORT No. 7226 
COMPUTER-BASED PROJECT for the EVALUATION 
of MEDIA for the HANDICAPPED 

T i t I G I EFFECTS OF MIXED AND UNMIXED CRITERION QUESTIONS 
BACKGROUND 

The Computer Based Project for the Evaluation of Media for the Handicapped, 
based on contract #OEC-9-423617-4357 (616) between the Syracuse (N.Y/) City School 
District and the Media Services and Captioned Fili^s feranch. Bureau of Education 
for the Handicapped (United States Office of Education) for the five year period 
July li 1969 through June 30, 1974. The major goal is to improve the instruction 
of handicapped children through the development and use of an evaluation system 
to measure the instructional effectiveness of films and other materials with 
educable mentally handicapped (KMH) children, in-service training and media suppor* 
for special teachers/ and studies related to the evaluation process and the 
populations used. 

The Project has concentrated on the 600 films and 200 filmstrips from the 
Media Services and Captioned Films (BEH • USOB) depositor^; however, specific 
packages from Project LIFE, various elementary math curricula, and selected 
programs from Children's TV Vtorkshop have also been evaluated. The evaluation 
model used requires that si) objectives of matftrials be specified and written; 

2) instruments be constructed to test and measure effectiveness; and ^ 3) children 
be the major sources of evaluation information. A number of instruments &nd- * 
methodologies are employed in the gathering of cognitive and affective data from 
900 EMH children and GO special teachers to raake the effectiveness decisions. 
Over half of the EMH population can neither read or write; therefore, a unique 
Student Response System (SKS) is employed, consisting of a twenty station G»E.- 
1000 SRS which can be operated in a group or individual recording mode and is 
connected to a remote computer system/ The computer irapabilities consist of 
remote telephone connections to the Rome (N.Y.) Air Development Command, the 
Honeywell time-shared network, and the Schenectady (N.Y. ) G B Research and 
Development Center; and batch mode capabilities of the Syracuse City Schools, 
Syracuse University, and various coir^mercial sources. 

In-service and media support activities provide on-the-job training for 
teachers, teacher aides, equipment, and materials to the special teachers in 
the city schools . The research activities have centered around investigations 
and special problems related to the development of the evaluation model. The 
four major areas considered are: ' 1) testing effects, 2) captioning effects, 

3) special student characteristics; and, 4) evaluation procedures validation. 

: D of the major activities appear in the five annual reports 

and the 600 evaluations prepared on materials used. Staff members v;ore encouraged 
to prepare special reports and the attached paper is one of these. The opinions 
OHpressed in this publication do not necessarily reflect the position or policy 
<>f ' the Dased Project, the United States Office of Education, or the 

liyir^c City School District, and no of ficial endorsement by any of the agencies 
should be inferred. . 
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EFFECTS OF MIXED AND UNMIXED CRITERION QUESTIONS 

At the Computer Based Project a procedure of using Independent 
samples to obtain base line responses to cognitive criterion Items 
has replaced one which used pretest/posttest data on the same sample 
for each film showing (CBP, 1972). The new procedure was Instituted 
to shorten the t!me required to administer quest Ion I tems ; It requires 
that Items from two films be Intermixed^ and presented after showing 
only one of the films. Thus the sample audience responds to questions 
from both the unseen film and the seen film after being exposed to 
only one f^lm to which the items app^y. 

The quest Ion has been ra I sedi - What effects have thetfe unrelated 
Items on those Items that are related to the f I Im seen?" The specific 
hypothesis Is there are no significant differences {P «» ,05) between 
the correct response of groups receiving only the I tems for the film 
seen and those receiving mixed Item sets* 

METHOD 

ft was observed that through an admin I stratlve error several sample 
audiences consisting of EMR and normal children In primary and Intermed* 
late ''Intact'^ classrooms were shown the f i Im /?1^26 V'Snlf fy Escapes 
Poisoning," and asked only those questions pertaining to the film seen 
from the scheduled mixed question set. , Other samples from the same 
population were shown the film and asked al! of the Items In the mixed 
question set, containing both Items for /?1^26 (the film seen) and for 



a film not seen. The strata for Items and for two academic groups 
■ were represented !n the available samples: educationally mentally 
'retarded (Group I, mixed Items, and Group II, o(\\y If \h2$ Items, and 
regular (normal CA 8-12) children Group 111, mixed items). Both 
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MIXED AND UJIillxiiD ITEIJS PERCBUT CORRECT AND RANKS FOR mi 
AND NOR>IAL SAIIPLES ON EILII tfU26 
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EI-Ul (GROUP I) 
(1) Mixed 



03 

A. Ranks I ill 15.5 

B. Ranks I & III 13.5 

C. Ranks I, II, & III 20.5 



mi (GROUP 11) 
(2) Untaixcd 



58 

D. Ranks X & II 10 

E. Ranko II i III 10 
V. Ranks I, II, & III 15 



MIXED (GROUP III) 
(3) Regular 



. : : : ■ 36 ; 

G. Ranks I & III 1.5 

11. Ranks II i 111 1.5 

I. Ranks '!, Hi * HI 3.5 
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Group I and III received the items In mixed order, and Group H re- 
ceived an unmixed set of items. 

The percent correct response for those Items for #l'i26 In these 
two situations Is summarized in Table I by percent of respondents 
giving a correct response to the . item and the rank of the score when 
combining the item per correct scores to rank order from lowest to 
highest in four comparisons I with It, II with Ml, III with I, 
and all three. For example, the ^'68'- for Item 1, Group i, in Table 

I is 15th when ranked with Group II scores; 13th when ranked with 
Group lit scores; and 20th when all three groups are ranked. 

The above item percentage scores Indicate a nominal order, con* 
tinuous distribution for|each sample groups The effects of mixing 

I I ems and d i f f erences b^ jween EMR a rid norma 1 popu lat 1 ons were tested 
using a 3 group KruskalfWal 1 Is one-way analysis of variance on the 
ranked item percent correct scores (Selgel, p. iB^-^lSSi 1956). The 
purpose was to test thefnull hypothesis that (Hq) there is no differ** 
ence between the average correct percent score of the three samples, 
and an al ternat Ive (H)) that the 3 samples are different in the aver- 
age percent correct scores. The value of the statistic H was computed 
from the sum of the ranks for each sample using the values from lines 
C, F, and 1 , of Table I above in the H formula (Setgel , p. I87) • The 
correction factor of .993 for seven pairs of tied values present In 
the data was also computed. The resulting H « 3.00 was appropriate 
for p « *30 which is greater than 'the rejection value of p « .05 
(Table C, Seigel, p. 2^9). Thus the null hypothesis cannot be reje^Cted 



and indicates that the sample groups are not significantly different 
In the average percent correct score* 

" " 12 f^MUl ^^y^^ ^ 1J79?2 . - 3(30 + 1) « 2,9819 

30 (30 + 1) ^ 10 + ^ToT"'*' 10 r 

Corrected H « H « 3,0012 

J I 2) + 15^ - 5] + 2 - 3] ^ 

( L30j3 - 30) 

The finding suggests that data from any one of these samples could 
be used to make item inclus ion dec is ions , or estimates of population 
responses, and that the responses of the groups are not significantly 
different from one another* It could be concluded that the effects of 
mixing items had no significant effect upon the resulting scores for 
items for the f I Im seen . 

V The differences in group means from Table I Indicate that the mixed 
sets for EHR chi Idren have' the lowest mean score, and that normal 
groups have the highest mean score. This finding is expected. It was 
assumed that a greater respondent frustration would be .expected from 
the mixture of relevant (about film seen) and 12 irrelevant (unseen 
film) items resulting in a ]ower correct score. A number of studies 
(CBP 711 • 718, 725i HIcro-ExpJ have suggested that normal groups do 
better than EMH groups academic tasks. 

The group mean differences were submitted to a Mann*Whitney U test 
(the non-parametric substitution for a t-tost of means when data Is 
ordinal), to determine Jf the differences were significant, A tabled 
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value for U « 27, p = .05, for groups of 10 is required to reject the 
null hypothesis that means are the same (Tahiti K, Selgel, I956, p, 277), 
An obtained value of between group comparisons was calculated from the 
ranks in Table I resulting in the following values: U^g " ''^j U23 «38^ 
U13 = U3. All of these values exceed the tabled value, and suggest the 
mean of each group is different from the other two groups. Adding this 
finding to the K-W finding of the samples from -ihe same population, in- 
dicates the samples did perform differently. 

If one uses a criterion of ^0% correct response on the item as 
acceptable (CBP, 1972) for Inclusion in the evaluation instrument, it 
can be noted that one item (#6) would be rejected if the mixed sample 
were used (Group I, Table I); two items (#5, #8) would be rejected by 
the unmixed sample (Group II, Table 1)} and two items (#1, #2) by the 
regular Group III. The inconsistency of the specific items that might 
be rejected seems to be due to unexplained sample variations; however, 
each would rej ect at least one item , The percentages across ^11 three 
groups for item #5 tend to be somewhat in the lower range of each set 
(i*th in Group I, 9th in Group II, and 6th in Group III); however, the 
responses for #8 and do not seem to fit any pattern;, in fact, a re- 
versal is evident. Item #1 is high for Group I and low for Group III; 
whereas, item #8 is high for Group III and low for Group II. 

This conclusion is further statistically verified by computing 
Spearman rank order correlations using the items by groups indicates 
the variability of item scores between groups. If each group received « 
the same proportion of the group score from each of the 'items the corre- 
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SPEARMAN RANK ORDER CORRELATION 



I 



II 



III 



Group I 



-.1*0 -.20 



Group II 



Note the correlations are all negative, suggesting the inverse 
ranking relationship between the groups noted in the paragraph above • 
Also the contributions of individual items to the total score of each 
sample group are not proportional and equally distributed. 

A graphic presentation of tKe dispersion of scores is shown below ^ 
suggesting the reason for the apparent Idlscrepancies between the K-W 
and Mann--Whitney findings as being one of dispersion of the scores* 
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IMPLICATIONS 

n ■ 

iThere seems to be an effect of mixing items which results in lower 
scores for EMR children on the criterion items for the film seen than 
when only items pertaining tp the film are asked* In a system where future 
decisions are made at the 50/? level, the item decisions tend to be on the 
conservative side thus resulting in the discarding of some good items, and 
probably results in lower achievement from mixed item setj? than would be 
indicated when only the items for the film seen are used* 
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